Data extraction methods for systematic review (semi)automation: Update of a living systematic review

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.


Introduction
In a systematic review, data extraction is the process of capturing key characteristics of studies in structured and standardised form based on information in journal articles and reports.It is a necessary precursor to assessing the risk of bias in individual studies and synthesising their findings.Interventional, diagnostic, or prognostic systematic reviews routinely extract information from a specific set of fields that can be predefined. 1 The most common fields for extraction in interventional reviews are defined in the PICO framework (population, intervention, comparison, outcome) and similar frameworks are available for other review types.The data extraction task can be time-consuming and repetitive when done by hand.This creates opportunities for support through intelligent software, which identify and extract information automatically.When applied to the field of health research, this (semi) automation sits at the interface between evidencebased medicine (EBM) and data science, and as described in the following section, interest in its development has grown in parallel with interest in AI in other areas of computer science.

Related systematic reviews and overviews
This review is, to the best of our knowledge, the only living systematic review (LSR) of data extraction methods.We identified four previous reviews of tools and methods in the first iteration of this living review (called base-review hereafter), [2][3][4][5] and two documents providing overviews and guidelines relevant to our topic. 3,6,710][11][12][13] Related reviews up to 2014: The systematic reviews from 2014 to 2015 present an overview of classical machine learning and natural language processing (NLP) methods applied to tasks such as data mining in the field of evidencebased medicine.At the time of publication of these documents, methods such as topic modelling (Latent Dirichlet Allocation) and support vector machines (SVM) were considered state-of-the art for language models.In 2014, Tsafnat et al. provided a broad overview on automation technologies for different stages of authoring a systematic review. 5O'Mara-Eves et al. published a systematic review focusing on text-mining approaches in 2015. 4t includes a summary of methods for the evaluation of systems, such as recall, accuracy, and F1 score (the harmonic mean of recall and precision, a metric frequently used in machine-learning).The reviewers focused on tasks related to PICO classification and supporting the screening process.In the same year, Jonnalagadda, Goyal and Huffman 3 described methods for data extraction, focusing on PICOs and related fields.The age of these publications means that the latest static or contextual embedding-based and neural methods are not included.These newer methods, 14 however, are used in contemporary systematic review automation software which will be reviewed in the scope of this living review.
Related reviews up to 2020: Reviews up to 2020 focus on discussions around tool development and integration in practice, and mark the starting date of the inclusion of automation methods based on neural networks.Beller et al.

UPDATE Amendments from Version 1
This version of the LSR includes 23 new papers, a change in the title indicates that the current version is an update.Ailbhe Finnerty and Rebecca Elmore joined the author team after contributing to screening and data extraction; Luke A. McGuinness contributed to the base-review but is not listed as an author in this update.The abstract and conclusions were updated to reflect changes and new research trends such as increased availability of datasets, source code, more papers describing relation extraction and summarisation.We updated existing figures and tables with the exception of Table 1 (pre-processing techniques), because reliance on pre-processing has decreased in recent years.Table 1 in the appendix was renamed as 'Table A1' to avoid confusion with Table 1 in the main text.
In the base-review we assessed the included publications based on a list of 17 items in the domains of reproducibility (3.4.1), transparency (3.4.2), description of testing (3.4.3), data availability (3.4.4), and internal and external validity (3.4.5).The list of items was reduced to six items for the update, more information about the removed items can be found in the methods section of this LSR.We still include the following items: • 3.4.2.2Is there a description of the dataset used and of its characteristics?• 3.4.2.4Is the source code available?• 3.4.3.2Are basic metrics reported (true/false positives and negatives)?• 3.4.4.1 Can we obtain a runnable version of the software based on the information in the publication?• 3.4.4.2 Persistence: Can data be retrieved based on the information given in the publication?• 3.4.5.1 Does the dataset or assessment measure provide a possibility to compare to other tools in the same domain?
Additionally, spreadsheets with all extracted data and updated figures are available as Appendix D.
Any further responses from the reviewers can be found at the end of the article describe principles for development and integration of tools for systematic review automation. 6Marshall and Wallace 7 present a guide to automation technology, with a focus on availability of tools and adoption into practice.They conclude that tools facilitating screening are widely accessible and usable, while data extraction tools are still at piloting stages or require a higher amount of human input.
A systematic review of machine-learning for systematic review automation, published in Portuguese in 2020, included 35 publications.The authors examined journals in which publications about systematic review automation are published, and conducted a term-frequency and citation analysis.They categorised papers by systematic review task, and provided a brief overview of data extraction methods. 2 Related reviews after 2020: These six reviews include and discuss end-user tools and cover different tasks across the SR workflow, including data extraction.Compared with this LSR, these reviews are broader in scope but have less included references on the automation of data extraction.Ruiz and Duffy 10 did a literature and trend analysis showing that the number of published references about SR automation is steadily increasing.Sundaram and Berleant 11 analyse 29 references applying text mining to different parts of the SR process and note that 24 references describe automation in study selection while research gaps are most prominent for data extraction, monitoring, quality assessment, and synthesis. 11Khalil et al. 9 include 47 tools and descriptions of validation studies in a scoping review, of which 8 are available end-user tools that mostly focus on screening, but also cover data extraction and risk of bias assessments.They discuss limitations of tools such as lack of generalisability, integration, funding, and limited performance or access. 9ierco Jimenez et al. 8 included 63 references in a mapping review of machine-learning to assist SRs during different workflow steps, of which 41 were available end-user tools for use by researchers without informatics background.In accordance with other reviews they describe screening as the most frequently automated step, while automated data extraction tools are lacking due to the complexity of the task.Zhang et al. 12 included 49 references on automation of data extraction fields such as diseases, outcomes, or metadata.They focussed on extraction from traditional Chinese medicine texts such as published clinical trial texts, health records, or ancient literature. 12Schmidt et al. 13 published a narrative review of tools with a focus on living systematic review automation.They discuss tools that automate or support the constant literature retrieval that is the hallmark of LSRs, while well-integrated (semi) automation of data extraction and automatic dissemination or visualisation of results between official review updates is supported by some, but less common.

Aim
We aim to review published methods and tools aimed at automating or (semi) automating the process of data extraction in the context of a systematic review of medical research studies.We will do this in the form of a living systematic review, keeping information up to date and relevant to the challenges faced by systematic reviewers at any time.
Our objectives in reviewing this literature are two-fold.First, we want to examine the methods and tools from the data science perspective, seeking to reduce duplicate efforts, summarise current knowledge, and encourage comparability of published methods.Second, we seek to highlight the added value of the methods and tools from the perspective of systematic reviewers who wish to use (semi) automation for data extraction, i.e., what is the extent of automation?Is it reliable?We address these issues by summarising important caveats discussed in the literature, as well as factors that facilitate the adoption of tools in practice.

Registration/protocol
This review was conducted following a preregistered and published protocol. 15PROSPERO was initially considered as platform for registration, but it is limited to reviews with health-related outcomes.Any deviations from the protocol have been described below.

Living review methodology
We are conducting a living review because the field of systematic review (semi) automation is evolving rapidly along with advances in language processing, machine-learning and deep-learning.
The process of updating started as described in the protocol 15 and is shown in Figure 1.In short, we will continuously update the literature search results, using the search strategies and methods described in the section 'Search' below.PubMed and arXiv search results are updated daily in a completely automated fashion via APIs.Articles from the dblp, ACL, and OpenAlex via EPPI-Reviewer are added every two months.All search results are automatically imported to our living review screening and data extraction web-application, which is described in the section 'Data collection and analysis' below.
The decision for full review updates is made every six months based on the number of new publications added to the review.For more details about this, please refer to the protocol or to the Cochrane living systematic review guidance.In between updates, the screening process and current state of the data extraction is visible via the living review website.

Eligibility criteria
• We included full text publications that describe an original NLP approach for extracting data related to systematic reviewing tasks.Data fields of interest (referred to here as entities or as sentences) were adapted from the Cochrane Handbook for Systematic Reviews of Interventions, 1 and are defined in the protocol. 15We included the full range of NLP methods (e.g., regular expressions, rule-based systems, machine learning, and deep neural networks).
• Publications must describe a full cycle of the implementation and evaluation of a method.For example, they must report training and at least one measure of evaluating the performance of a data extraction algorithm.
• We included reports published from 2005 until the present day, similar to previous work. 3We would have translated non-English reports, had we found any.
• The data that the included publications use for mining must be texts from randomised controlled trials, comparative cohort studies, case control studies or comparative cross-sectional studies (e.g., for diagnostic test accuracy).The scope of data extraction methods can be applied to the full text or to abstracts within each eligible publication's corpus.We included publications that extracted data from other study types, as long as at least one of our study types of interest was contained in the corpus.
We excluded publications reporting: • Methods and tools related solely to image processing and importing biomedical data from PDF files without any NLP approach, including data extraction from graphs.
• Any research that focuses exclusively on protocol preparation, synthesis of already extracted data, write-up, solely the pre-processing of text or its dissemination.• Methods or tools that provided no natural language processing approach and offered only organisational interfaces, document management, databases, or version control • Any publications related to electronic health reports or mining genetic data.

Search
Base-review: We searched five electronic databases, using the search methods previously described in our protocol. 15n short, we searched MEDLINE via Ovid, using a search strategy developed with the help of an information specialist, and searched Web of Science Core Collection and IEEE using adaptations of this strategy, which were made by the review authors.Searches on the arXiv (computer science) and dblp were conducted on full database dumps using the search functionality described by McGuinness and Schmidt. 16The full search results and further information about document retrieval are available in Underlying data: Appendix A and B. 127 Originally, we planned to include a full literature search from the Web of Science Core Collection.Due to the large number of publications retrieved via this search (n = 7822) we decided to first screen publications from all other sources, to train a machine-learning ensemble classifier, and to only add publications that were predicted as relevant for our living review.This reduced the Web of Science Core Collection publications to 547 abstracts, which were added to the studies in the initial screening step.The dataset, code and weights of trained models are available in Underlying data: Appendix C. 127 This includes plots of each model's evaluation in terms of area under the curve (AUC), accuracy, F1, recall, and variance of cross-validation results for every metric.
Update: As planned, we changed to the PubMed API for searching MEDLINE.This decision was made to facilitate continuous reference retrieval.We searched only for pre-print or published literature and therefore did not search sources such as GITHUB or other source code repositories.
Update: We searched PubMed via its API, arXiv (computer science), ACL-Anthology, dblp, and used EPPI-Reviewer to collect citations from MicrosoftAcademic and later OpenAlex using the 'Bi-Citation AND Recommendations' method.

Selection of studies
Initial screening and data extraction were conducted as stated in the protocol.In short, for the base-review we screened all retrieved publications using the Abstrackr tool.All abstracts were screened by two independent reviewers.Conflicting judgements were resolved by the authors who made the initial screening decisions.Full texts screening was conducted in a similar manner to abstract screening but used our web application for LSRs described in the following section.
For the updated review we used our living review web application to retrieve all publications with the exception of the items retrieved by EPPI-Reviewer (these are added to the dataset separately).We further used our application to de-duplicate, screen, and data-extract all publications.
A methodological update to the screening process included a change to single-screening to assess eligibility on both abstract and full-text level, reducing dual-screening to 10% of the publications.

Data extraction, assessment, and management
We previously developed a web application to automate reference retrieval for living review updates (see Software availability 17 ), to support both abstract and full text screening for review updates, and to manage the data extraction process throughout. 17For future updates of this living review we will use the web application, and not Abstrackr, for screening references.This web application is already in use by another living review. 18It automates daily reference retrieval from the included sources and has a screening and data extraction interface.All extracted data are stored in a database.Figures and tables can be exported on a daily basis and the progress in between review updates is shared on our living review website.The full spreadsheet of items extracted from each included reference is available in the Underlying data. 127As previously described in the protocol, quality of reporting and reproducibility was initially assessed based on a previously published checklist for reproducibility in text mining, but some of the items were removed from the scope of this review update. 19 planned in the protocol, a single reviewer conducted data extraction, and a random 10% of the included publications were checked by a second reviewer.

Visualisation
The creation of all figures and interactive plots on the living review website and in this review's 'Results' section was automated based on structured content from our living review database (see Appendix A and D, Underlying data 127 ).We automated the export of PDF reports for each included publication.Calculation of percentages, export of extracted text, and creation of figures was also automated.

Accessibility of data
All data and code are free to access.A detailed list of sources is given in the 'Data availability' and 'Software availability' sections.

Changes from protocol and between updates
In the protocol we stated that data would be available via an OSF repository.Instead, the full review data are available via the Harvard Dataverse, as this repository allows us to keep an assigned DOI after updating the repository with new content for each iteration of this living review.We also stated that we would screen all publications from the Web of Science search.Instead, we describe a changed approach in the Methods section, under 'Search'.For review updates, Web of Science was dropped and replaced with OpenAlex searches via EPPI-Reviewer.
We added a data extraction item for the type of information which a publication mines (e.g.P, IC, O) into the section of primary items of interest, and we moved the type of input and output format from primary to secondary items of interest.We grouped the secondary item of interest 'Other reported metrics, such as impacts on systematic review processes (e.g., time saved during data extraction)' with the primary item of interest 'Reported performance metrics used for evaluation'.
The item 'Persistence: is the dataset likely to be available for future use?' was changed to: 'Can data be retrieved based on the information given in the publication?'.We decided not to speculate if a dataset is likely to be available in the future and chose instead to record if the dataset was available at the time when we tried to access it.
The item 'Can we obtain a runnable version of the software based on the information in the publication?' was changed to 'Is an app available that does the data mining, e.g. a web-app or desktop version?'.
In this current version of the review we did not yet contact the authors of the included publications.This decision was made due to time constraints, however reaching out to authors is planned as part of the first update to this living review.
In the base-review we assessed the included publications based on a list of 17 items in the domains of reproducibility (3.4.1), transparency (3.4.2), description of testing (3.4.3), data availability (3.4.4), and internal and external validity (3.4.5).The list of items was reduced to six items for the update: • 3.4.2.2Is there a description of the dataset used and of its characteristics?
• 3.4.4.1 Can we obtain a runnable version of the software based on the information in the publication?
• 3.4.4.2 Persistence: Can data be retrieved based on the information given in the publication?
• 3.4.5.1 Does the dataset or assessment measure provide a possibility to compare to other tools in the same domain?
The following items were removed, although the results and discussion from the assessment of these items in the basereview remains within the review text: • 3.4.1.1Are the sources for training/testing data reported?
• 3.4.2.1 Is there a description of the algorithms used?
• 3.4.3.1 Is there a justification/an explanation of the model assessment?
• 3.4.3.3Does the assessment include any information about trade-offs between recall or precision (also known as sensitivity and positive predictive value)?
• 3.4.4.3Is the use of third-party frameworks reported and are they accessible?
• 3.4.5.2 Are explanations for the influence of both visible and hidden variables in the dataset given?
• 3.4.5.5 Is the model's adaptability to different formats and/or environments beyond training and testing data described?

Results of the search
Our database searches identified 10,107 publications after duplicates were removed (see Figure 2).We identified one more publication manually.
This iteration of the living review includes 76 publications, summarised in Table A1 in Underlying data 127 ).

Excluded publications
Across the base-review and the update, 216 publications were excluded at the full text screening stage, with the most common reason for exclusion being that it did not fit target entities or target data.In most cases, this was due to the texttypes mined in the publications.Electronic health records and non-trial data were common, and we created a list of datasets that would be excluded in this category (see more information in Underlying data: Appendix B 127 ).Some publications addressed the right kind of text but were excluded for not mining data of interest to this review.For example, Norman, Leeflang and Névéol 23 performed data extraction for diagnostic test accuracy reviews, but focused on extracting the results and data for statistical analyses.Millard, Flach and Higgins 24 and Marshall, Kuiper and Wallace 25 looked at risk of bias classification, which is beyond the scope of this review.Boudin, Nie and Dawes 26 developed a weighing scheme based on an analysis of PICO element locations, leaving the detection of single PICO elements for future work.Luo et al. 27 extracted data from clinical trial registrations but focused on parsing inclusion criteria into event or temporal entities to aid participant selection for randomised controlled trials (RCTs).
The second most common reason for study exclusion was that they had 'no original data extraction approach'.Rathbone et al., 28 for example, used hand-crafted Boolean searches specific to a systematic review's PICO criteria to support the screening process of a review within Endnote.We classified this article as not having any original data extraction approach because it does not create any structured outputs specific to P, IC, or O. Malheiros et al. 29 performed visual text mining, supporting systematic review authors by document clustering and text highlighting.Similarly, Fabbri et al. 30 implemented a tool that supports the whole systematic review workflow, from protocol to data extraction, performing clustering and identification of similar publications.Other systematic reviewing tasks that can benefit from automation but were excluded from this review are listed in Underlying data: Appendix B. 127 3.2 Results from the data extraction: Primary items of interest

Automation approaches used
Figure 3 shows aspects of the system architectures implemented in the included publications.A short summary of these for each publication is provided in Table A1 in Underlying data. 127Where possible, we tried to break down larger system architectures into smaller components.For example, an architecture combining a word embedding + long shortterm memory (LSTM) network would have been broken down into the two respective sub-components.We grouped binary classifiers, such as naïve Bayes and logistic regression.Although SVM is also binary classifier, it was assigned as separate category due to its popularity.The final categories are a mixture of non-machine-leaning automation (application programming interface (API) and metadata retrieval, PDF extraction, rule-base), classic machine-learning (naïve Bayes, decision trees, SVM, or other binary classifiers) and neural or deep-learning approaches (convolutional neural network (CNN), LSTM, transformers, or word embeddings).This figure shows that there is no obvious choice of system architecture for this task.For the LSR update, the strongest trend was the increasing application of BERT (Bidirectional Encoder Representations from Transformers).BERT was published in 2018 and other architecturallyidentical versions of it tailored to using scientific text, such as SciBERT, are summarised under the same category in this review. 14,31In the base-review, BERT was used three times, whilst now appearing 21 times.[36] Figure ).Although used more frequently in the past, the 11 publications published between 2017 and now that use this approach alongside other architectures such as BERT, 33,[37][38][39] conditional random fields (CRF), 40 use it with SVM 41 or other binary classifiers. 42In practice, these systems use rule-bases in the form of hand-crafted lists to identify candidate phrases for amount entities such as sample size 42,43 or to refine a result obtained by a machine-learning classifier on the entity level (e.g., instances where a specific intervention or outcome is extracted from a sentence). 40nary classifiers, most notably naïve Bayes and SVMs, are also frequently used system components in the data extraction literature.They are frequently used in studies published between 2005 and now but their usage started declining with the advent of neural models.
Embedding and neural architectures are increasingly being used in literature over the past seven years.Recurrent neural networks (RNN), CNN, and LSTM networks require larger amounts of training data; by using transformer-based embeddings with pre-training algorithms based on unlabelled data they have become increasingly more interesting in fields such as data extraction for EBM-where high-quality training data are difficult and expensive to obtain.
In the 'Other' category, tools mentioned were mostly other classifiers such as maximum entropy classifiers (n = 3), kLog, J48, and various position or document-length classification algorithms.We also added methods such as supervised distant supervision (n = 3, see Ref. 44) and novel training approaches to existing neural architectures in this category.

Reported performance metrics used for evaluation
Precision (i.e., positive predictive value), recall (i.e., sensitivity), and F1 score (harmonic mean of precision and recall) are the most widely used metrics for evaluating classifiers.This is reflected in Figure 4, which shows that at least one of these metrics was used in the majority of the included publications.Accuracy and area under the curve -receiver operator characteristics (AUC-ROC) were less frequently used.There were several approaches and justifications of using macro-or micro-averaged precision, recall, or F1 scores in the included publications.Micro or macro scores are computed in multi-class cases, and the final scores can differ whenever the classes in a dataset are imbalanced (as is the case in most datasets used for automating data extraction in SR automation).
Both micro and macro scores were reported by Singh et al. (2021), 45 48,49 reported micro across documents, and macro across the classes.
Macro-scores were used in one publication. 37cro scores were used by Fiszman et al. 47 for class-level results.In one publication harmonic mean was used for precision and recall, while micro-scoring was used for F1. 50Micro scores were most widely used, including Al-Hussaini et al. ( 2022), 32 Sanchez-Graillet et al. (2022), 51 Kim et al. (2011), 52 Verbeke et al. ( 2012), 53 and Jin and Szolovits (2020) 54 were used in the evaluation script of Nye et al. (2018). 55 the category 'Other' we added several instances where a relaxation of a metric was introduced, e.g., precision using topn classified sentences 44,46,56 or mean average precision and the metric 'precision @rank 10' for sentence ranking exercises. 57,58Another type of relaxation for standard metrics is a distance relaxation when normalising entities into concepts in medical subject headings (MesH) or unified medical language system (UMLS), to allow N hops between predicted and target concepts. 59e LSR update showed an increasing trend of text summarisation and relation extraction algorithms.ROGUE, ΔEI, or Jaccard similarity were metrics for summarisation. 60,61For relation extraction F1, precision, and recall remained the most common metrics. 62,63her metrics were kappa, 58 random shuffling 64 or binomial proportion test 65 to test statistical significance, given with confidence intervals. 41Further metrics included under 'Other' were odds ratios, 66 normalised discounted cumulative gain, 44,67 'sentences needed to screen per article' in order to find one relevant sentence, 68 McNemar test, 65 C-statistic (with 95% CI) and Brier score (with 95% CI). 69Barnett (2022) 70 extracted sample sizes and reported the mean difference between true and extracted numbers.
Real-life evaluations, such as the percentage of outputs needing human correction, or time saved per article, were reported by two publications, 32,46 and an evaluation as part of a wider screening system was done in another. 71

Scope and data
Most data extraction is carried out on abstracts (See Table A1 in Underlying data, 127 and the supplementary table giving an overview of all included publications).Abstracts are the most practical choice, due to the possibility of exporting them along with literature search results from databases such as MEDLINE.In total, 84% (N=64) of the included publications directly reported using abstracts.Within the 19 references (25%) that reported usage of full texts, eight specifically mentioned that this also included abstracts but it is unclear if all full texts included abstract text.Descriptions of the benefits of using full texts for data extraction include having access to a more complete dataset, while the benefits of using titles (N=4, 5%) include lower complexity for the data extraction task. 43Xu et al. ( 2010) 72 exclusively used titles, while the other three publications that specifically mentioned titles also used abstracts in their datasets. 43,73,74gure 5 shows that RCTs are the most common study design texts used for data extraction in the included publications (see also extended Table A1 in Underlying data 127 ).This is not surprising, because systematic reviews of interventions are the most common type of systematic review, and they are usually focusing on evidence from RCTs.Therefore, the literature for automation of data extraction focuses on RCTs, and their related PICO elements.Systematic reviews of diagnostic test accuracy are less frequent, and only one included publication specifically focused on text and entities related to these studies, 75 while two mentioned diagnostic procedures among other fields of interest. 35,7676,77 More publications mining data from surveys, animal RCTs, or case series might have been found if our search and review had concentrated on these types of texts.

Data extraction targets
Mining P, IC, and O elements is the most common task performed in the literature of systematic review (semi-) automation (see Table A1 in Underlying data, 127 and Figure 6).In the base-review, P was the most common entity.
After the LSR update, O (n=52, 68%) has become the most popular, due to the emerging trend of relation-extraction models that focus on the relationship between O and I entities and therefore may omit the automatic extraction of P. Some of the less-frequent data extraction targets in the literature can be categorised as sub-classes of a PICO, 55 for example, by annotating hierarchically multiple entity types such as health condition, age, and gender under the P class.3][84] Usually, I and C are merged (n=47).Most data extraction approaches focused on recognising instances of entity or sentence classes, and a small number of publications went one step further to normalise to actual concepts and including data sources such as UMLS (Unified Medical Language System). 35,39,59,73,85e 'Other' category includes some more detailed drug annotations 65 or information such as confounders 49 and other entity types (see the full dataset in Underlying data: Appendix A and D for more information 127 ).

Results from the data extraction: Secondary items of interest 3.3.1 Granularity of data extraction
A total of 54 publications (71%) extracted at least one type of information at the entity level, while 46 publications (60%) used sentence level (see Table A1 extended version in Underlying data 127 ).We defined the entity level as any number of words that is shorter than a whole sentence, e.g., noun-phrases or other chunked text.Data types such as P, IC, or O commonly appeared to be extracted on both entity and sentence level, whereas 'N', the number of people participating in a study, was commonly extracted on entity level only.

Type of input
The majority of publications and benchmark corpora mentioned MEDLINE, via PubMed, as the data source for text.Text files (n = 64), next to XML (n = 8), or HTML (n = 3), are the most common format of the data downloaded from these sources.Therefore, most systems described using, or were assumed to use, text files as input data.Eight included publications described using PDF files as input. 44,46,59,68,75,81,86,87

Type of output
A limited number of publications described structured summaries as output of their extracted data (n = 14, increasing trend between LSR updates).Alternatives to exporting structured summaries were JSON (n = 4), XML, and HTML (n = 2 each).Two publications mentioned structured data outputs in the form of an ontology. 51,88Most publications mentioned only classification scores without specifying an output type.In these cases, we assumed that the output would be saved as text files, for example as entity span annotations or lists of sentences (n = 55).

Assessment of the quality of reporting
In the base-review we used a list of 17 items to investigate reproducibility, transparency, description of testing, data availability, and internal and external validity of the approaches in each publication.The maximum and minimum number of items that were positively rated were 16 and 1, respectively, with a median of 10 (see Table A1 in Underlying data 127 ).Scores were added up and calculated based on the data provided in Appendix A and D (see Underlying data 127 ), using the sum and median functions integrated in Excel.Publications from recent years up to 2021 showed a trend towards more complete and clear reporting.Of the included publications in the base-review, 50 out of 53 (94%) clearly stated the sources of their data used for training and evaluation.MEDLINE was the most popular source of data, with abstracts usually described as being retrieved via searches on PubMed, or full texts from PubMed Central.A small number of publications described using text from specific journals such as PLoS Clinical Trials, New England Journal of Medicine, The Lancet, or BMJ. 56,83Texts and metadata from Cochrane, either provided in full or retrieved via PubMed, were used in five publications. 57,59,68,75,86orpora such as the ebm-nlp dataset, 55 or PubMed-PICO 54 are available for direct download.Publications published in recent years are increasingly reporting that they are using these benchmark datasets rather than creating and annotating their own corpora (see 4 for more details).After the publication of the base-review, transformer models such as BERT became dominant in the literature (see Figure 3).With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31We are therefore not going to update this table in this, or any future iterations of this LSR.We leave it for reference to publications that may still use these methods in the future.

Transparency of methods
3.4.2.1 Is there a description of the algorithms used?
Figure 7 shows that 43 out of 53 publications in the base-review (81%) provided descriptions of their data extraction algorithm.In the case of machine learning and neural networks, we looked for a description of hyperparameters and feature generation, and for the details of implementation (e.g. the machine-learning framework).Hyperparameters were rarely described in full, but if the framework (e.g., Scikit-learn, Mallet, or Weka) was given, in addition to a description of implementation and important parameters for each classifier, then we rated the algorithm as fully described.For rule-based methods we looked for a description of how rules were derived, and for a list of full or representative rules given as examples.Where multiple data extraction approaches were described, we gave a positive rating if the bestperforming approach was described.
3.4.2.2Is there a description of the dataset used and of its characteristics?
Of the included publications in the review update, 73 out of 76 (97%) provided descriptions of their dataset and its characteristics.
Most publications provided descriptions of the dataset(s) used for training and evaluation.The size of each dataset, as well as the frequencies of classes within the data, were transparent and described for most included publications.All dataset citations, along with a short description and availability of the data, are shown in Table 4.
Most included publications in the base-review did not report their hardware specifications, though five publications (9%) did.One, for example, applied their system to new, unlabelled data and reported that classifying the whole of PubMed takes around 20 hours using a graphics processing unit (GPU). 69In another example, the authors reported using Google Colab GPUs, along with estimates of computing time for different training settings. 954.2.4Is the source code available?
Figure 8 shows that most of the included publications did not provide any source code, although there is a very strong trend towards better code-availabilty in the publications from the review update (n=19 published code, 83% of the new publications provided code).Publications that did provide the source code were exclusively published or last updated in the last seven years.GitHub is the most popular platform for making code accessible.Some publications also provided links to notebooks on Google Colab, which is a cloud-based platform to develop and execute code online.Two publications provided access to parts of the code, or access was restricted.A full list of code repositories from the included publications is available in Table 2.    Of the included publications in the base-review, 47 out of 53 (89%) gave a detailed assessment of their data extraction algorithms.We rated this item as negative if only the performance scores were given, i.e., if no error analysis was performed and no explanations or examples were given to illustrate model performance.In most publications a brief error analysis was common, for example discussions on representative examples for false negatives and false positives, 47 major error sources 90 or highlighting errors with respect to every entity class. 76Both Refs.52, 53 used structured and unstructured abstracts, and therefore discussed the implications of unstructured text data for classification scores.
A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets or tested while conducting actual systematic reviews. 46,58,63,69,48,95,101,1024.3.2Are basic metrics reported (true/false positives and negatives)?
Figure 9 shows the extent to which all raw basic metrics, such as true-positives, were reported in the included publications in the LSR update.In most publications (n = 62) these basic metrics are not reported, and there is a trend between basereview and this update towards not reporting these.However, basic metrics could be obtained since the majority of new included publications made source code available and used publicly available datasets.When dealing with entity-level data extraction it can be challenging to define the quantity of true negative entities.This is true especially if entities are labelled and extracted based on text chunks, because there can be many combinations of phrases and tokens that constitute an entity. 47This problem was solved in more recent publications by conducting a token-based evaluation that computes scores across every single token, hence gaining the ability to score partial matches for multi-word entities. 554.3.3Does the assessment include any information about trade-offs between recall or precision (also known as sensitivity and positive predictive value)?
Of the included publications in the base-review, 17 out of 53 (32%) described trade-offs or provided plots or tables showing the development of evaluation scores if certain parameters were altered or relaxed.Recall (i.e., sensitivity) is often described as the most important metric for systematic review automation tasks, as it is a methodological demand that systematic reviews do not exclude any eligible data.
References 56 and 76 showed how the decision of extracting the top two or N predictions impacts the evaluation scores, for example precision or recall.Reference 102 shows precision-recall plots for different classification thresholds.
Reference 72 shows four cut-offs, whereas Ref. 95 shows different probability thresholds for their classifier, and describe the impacts of this on precision, recall, and F1 curves.
Some machine-learning architectures need to convert text into features before performing classification.A feature can be, for example, the number of times that a certain word occurs, or the length of an abstract.The number of features used, e.g. for CRF algorithms, which was given in multiple publications, 92 together with a discussion of classifiers that should be used in high recall is needed. 42,103show ROC curves quantifying the amount of training data and its impact on the scores.Compiling and testing code from every publication is outside the scope of this review.Instead, in Figure 10 and Table 3 we recorded the publications where a (web) interface or finished application was available.Counting RobotReviewer and Trialstreamer as separate projects, 12% of the included publications had an application associated with it, but only 5 (6%) are available and directly usable via web-apps.Applications were available as open-source, completely free, or free basic versions with optional features that can be purchased or subscribed to.
3.4.4.2 Persistence: Can data be retrieved based on the information given in the publication?
We observed an increasing trend of dataset availability and publications re-using benchmark corpora within the LSR update.Only seven of the included publications in the base-review (13%) made their datasets publicly available, out of the 36 unique corpora found then.
After the LSR update we accumulated 55 publications that describe unique new corpora.Of these, 23 corpora were available online and a total of 40 publication mentioned using one of these public benchmarking sets.Table 4 shows a summary of the corpora, their size, classes, links to the datasets, and cross-reference to known publications re-using each data set.For the base review, we collected the corpora, provide a central link to all datasets, and will add datasets as they become available during the life span of this living review (see Underlying data 127,128 below).Due to the increased number of available corpora we stopped downloading the data and provide links instead.When a dataset is made freely available without barriers (i.e., direct downloads of text and labels), then any researcher can re-use the data and publish results from different models, which become comparable to one another.Copyright issues surrounding data sharing were Of the included publications in the base-review, 47 out of 53 (88%) described using at least one third-party framework for their data extraction systems.The following list is likely to be incomplete, due to non-available code and incomplete reporting in the included publications.Most commonly, there was a description of machine-learning toolkits (Mallet, N = 12; Weka, N = 6; tensorflow, N = 5; scikit-learn, N = 3).Natural language processing toolkits such as Stanford parser/ CoreNLP (N = 12) or NLTK (N = 3), were also commonly reported for the pre-processing and dependency parsing steps within publications.The MetaMap tool was used in nine publications, and the GENIA tagger in four.For the complete list of frameworks please see Appendix A and D in Underlying data.With this item we aimed to assess publications to see if the evaluation results from models are comparable with the results from other models.Ideally, a publication would have reported the results of another classification model on the same dataset, either by re-implementing the model themselves 96 or by describing results of other models when using benchmark datasets. 64This was rarely the case for the publications in the base-review, as most datasets were curated and used in single publications only.However, the re-use of benchmark corpora increased with the publications in the LSR update, where we found 40 publications that report results on one of the previously published benchmark datasets (see Table 4).
Addtionally, in the base-review, in 40 publications (75%) data were well described, and they utilised commonly used entities and common assessment metrics, such as precision, recall, and F1-scores, leading to a limited comparability of results.In these cases, the comparability is limited because those publications used different data sets, which can influence the difficulty of the data extraction task and lead to better results within for example structured datasets or topicspecific datasets.
3.4.5.2 Are explanations for the influence of both visible and hidden variables in the dataset given?
This item relates only to publications using machine learning or neural networks.Rule-based classification systems (N = 8, 15% reporting rule-base as sole approach) are not applicable to this item, because the rules leading to decisions are intentionally chosen by the creators of the system and are therefore always visible.
Ten publications in the base-review (19%) discussed hidden variables. 83discussed that the identification of the treatment group entity yielded the best results.However, when neither the words 'group' nor 'arm' were present in the text then the system had problems with identifying the entity.'Trigger tokens' 104 and the influence of common phrases were also described by Ref. 68, the latter showed that their system was able to yield some positive classifications in the absence of common phrases. 103went a step further and provided a table with words that had the most impact on the prediction of each class. 57describes removing sentence headings in structured abstracts in order to avoid creating a system biased towards common terms, while Ref. 90 discussed abbreviations and grammar as factors influencing the results.Length of input text 59 and position of a sentence within a paragraph or abstract, e.g. up to 10% lower classification scores for certain sentence combinations in unstructured abstracts, were shown in several publications. 46,66,1024.5.3Is the process of avoiding overfitting or underfitting described?
'Overfitted' is a term used to describe a system that shows particularly good evaluation results on a specific dataset because it has learned to classify noise and other intrinsic variations in the data as part of its model. 105 the included publications in the base-review, 33 out of 53 (62%) reported that they used methods to avoid overfitting.Eight (15%) of all publications reported rule-based classification as their only approach, allowing them to not be susceptible to overfitting by machine learning.
Furthermore, 28 publications reported cross-validation to avoid overfitting.Mostly these classifiers were in the domain of machine-learning, e.g.SVMs.Most commonly, 10 folds were used (N = 15), but depending on the size of evaluation corpora, 3, 6, 5 or 15 folds were also described.Two publications 55,85 cautioned that cross-validation with a high amount of folds (e.g.10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.One publication 104 stratified folds by class in order to avoid this variance in evaluation results in a fold which is caused by a sparsity of positive instances.
Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. 59,96,106Some publications did not specifically discuss overfitting in the text, but their open-source code indicated that the latter techniques were used. 55,75Random allocation to treatment groups is an important item when assessing bias in RCTs, because selective allocation can lead to baseline differences. 1Similarly the process of splitting a dataset randomly, or in a stratified manner, into training (or rule-crafting) and test data is important when constructing classifiers and intelligent systems. 117l included publications in the base-review gave indications of how different train and evaluation datasets were obtained.Most commonly there was one dataset and the splitting ratio which indicated that splits were random.This information was provided in 36 publications (68%).
For publications mentioning cross-validation (N = 28, 53%) we assumed that splits were random.The ratio of splitting (e.g.80:20 for training and test data) was clear in the cross-validation cases and was described in the remainder of publications.
It was also common for publications to use completely different datasets, or multiple iterations of splitting, training and testing (N = 13, 24% For this item we aimed to find out how many of the included publications in the base-review tested their data extraction algorithms on different datasets.A limitation often noted in the literature was that gold-standard annotators have varying styles and preferences, and that datasets were small and limited to a specific literature search.Evaluating a model on multiple independent datasets provides the possibility of quantifying how well data can be extracted across domains and how flexible a model is in real-life application with completely new data sets.Of the included publications, 19 (36%) discussed how their model performed on datasets with characteristics that were different to those used for training and testing.In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets.These are further discussed in the 'Discussion' section of this living review.

Sources of funding and conflict of interest
Figure 11 shows that most of the included publications in the base review did not declare any conflict of interest.This is true for most publications published before 2010, and about 50% of the literature published in more recent years.However, sources of funding were declared more commonly, with 69% of all publications including statements for this item.This reflects a trend of more complete reporting in more recent years.

Evaluation
We found that precision, recall, and F1 were used as evaluation metrics in most publications, although sometimes these metrics were adapted or relaxed in order to account for partial or similar matches.

Scope
Most of the included publications focused on extracting data from abstracts.The reasons for this include the availability of data and ease of access, as well as the high coverage of information and the availability of structured abstracts that can automatically derive labelled training data.A much smaller number of the included publications (n=19, 25%) extracted data from full texts.Half of the systems that extract data from full text were published within the last seven years.In systematic review practice, manually extracting data from abstracts is quicker and easier than manually extracting data from full texts.Therefore, the potential time-saving and utility of full text data extraction is much higher because more time can be saved by automation and it provides automation that more closely reflects the work done by systematic reviewers in practice.However, the data extraction literature on full text is still sparse and extraction from abstracts may be of limited value to reviewers in practice because it carries the risk of missing information.Whenever a publication reported full-text extraction we tried to find out if this also included abstract text, in which case we would count the publication in both categories.However, this information was not always clearly reported.

Target texts
Reports of randomised controlled trials were the most common texts used for data extraction.Evidence concerning data extraction from other study types was rare and is discussed further in the following sections.

Assessment of the quality of reporting
We only assessed full quality of reporting in the base-review, and assessed selected items during the review update.The quality of reporting in the included studies in the base-review is improving over time We assessed the included publications based on a list of 17 items in the domains of reproducibility, transparency, description of testing, data availability, and internal and external validity.Base-review: Reproducibility was high throughout, with information about sources of training and evaluation data reported in 94% of all publications and pre-processing described in 89%.
Base-review: In terms of transparency, 81% of the publications provided a clear description of their algorithm, 94% described the characteristics of their datasets, but only 9% mentioned hardware specifications or feasibility of using their algorithm on large real-world datasets such as PubMed.
Update: Availability of source code was high in the publications added in the LSR update (N=19, 83%).Before the update, 15% of all included publications had made their code available.Overall, 39% (N=30) now have their code available and all links to code repositories are shown in Table 2.
Base-review: Testing of the systems was generally described, 89% gave a detailed assessment of their algorithms.Tradeoffs between precision and recall were discussed in 32%.
Update: Basic metrics were reported in only 19% (N=14) of the included publications, which is a downward trend from 24% in the base-review.However, more complete reporting of source-code and public datasets still leads to increased transparency and comparability.
Update: Availability of the final models as end-user tools was very poor.Only 12% of the included publications had an application associated with it, but only 5 (6%) are available and directly usable via web-apps (see Table 3 for links).Furthermore, it is unclear how many of the other tools described in the literature are used in practice, even if only used internally within their authors research groups.There was a surprisingly strong trend towards sharing and re-using already published corpora in the LSR update.Earlier, labelled training and evaluation data were available from 13% of the publications, and only a further 32% of all publications reported using one of these available datasets.Within the LSR update, 22 corpora were available online and at least 40 other included publication mention using them.Table 4 provides the sources of all corpora and publications using them.For named-entity recognition, EBM-NLP 55 is the most popular dataset, used by at least 10 other publications and adapted and used by another four.For sentence classification the NICTA gold-standard 52 is used by eight others, and the automatically labelled corpus by Jin and Szolovits 96 is used by five others and was adapted once.For relation extraction the EvidenceInference 2.0 corpus is gaining attention, being used in at least three other publications.
Base-review: A total of 88% of the publications described using at least one accessible third-party framework for their data extraction system.Internal and external validity of each model was assessed based on its comparability to other tools (75%), assessment of visible and hidden variables in the data (19%), avoiding overfitting (62%, not applicable to nonmachine learning systems), descriptions of splitting training from validation data (100%), and adaptability and external testing on datasets with different characteristics (36%).These items, together with caveats and limitations noted in the included publications are discussed in the following section.

Caveats and challenges for systematic review (semi)automation
In the following section we discuss caveats and challenges highlighted by the authors of the included publications.We found a variety of topics discussed in these publications and summarised them under seven different domains.Due to the increasing trend of relation-extraction and text summarisation models we now summarise any challenges or caveats related to these within the updated text at the end of each applicable domain.

Label-quality and inter-annotator disagreements
The quality of labels in annotated datasets was identified as a problem by several authors.The length of the entity being annotated, for example O or P entities, often caused disagreements between annotators. 46,48,58,69,95,101,102We created an example in Figure 12, which shows two potentially correct, but nevertheless different annotations on the same sentence.Similar disagreements, 65,85,104 along with missed annotations, 72 are time-intensive to reconciliate 97 and make the scores less reliable. 95As examples of this, two publications observed that their system performed worse on classes with high disagreement. 75,104There exist different explanations for worse performance in these cases.It is possibly harder for models to learn from labelled data with systematic differences within.Another reason is that the model learns predictions based on one annotation style and therefore artificial errors are produced when evaluated against differently labelled data, or that the annotation task itself is naturally harder in cases with high inter-annotator disagreement, and therefore lower performance from the models might be explainable.An overview of the included publications discussing this, together with their inter-annotator disagreement scores, is given in Table 5.
To mitigate these problems, careful training and guides for expert annotators are needed. 58,77For example, information should be provided on whether multiple basic entities or one longer entity annotation are preferred. 85Crowd-sourced annotations can contain noisy or incorrect information and have low interrater reliability.However, they can be aggregated to improve quality. 55In recent publications, partial entity matches (i.e., token-wise evaluation) downstream were generally favoured above complete detection, which helps to mitigate this problem's impact on final evaluation scores. 55,83or automatically labelled or distantly supervised data, label quality is generally lower.This is primarily caused by incomplete annotation due to missing headings, or by ambiguity in sentence data, which is discussed as part of the next domain. 44,57,103

Ambiguity
The most common source of ambiguity in labels described in the included publications is associated with automatically labelled sentence-level data.Examples of this are sentences that could belong to multiple categories, e.g., those that should have both 'P' and an 'I' label, or sentences that were assigned to the class 'other' while containing PICO information (Refs.54, 95, 96, among others).Ambiguity was also discussed with respect to intervention terms 76 or when distinguishing between 'control' and 'intervention' arms. 46When using, or mapping to UMLS concepts, ambiguity was discussed in Refs.41, 52, 72.
At the text level, ambiguity around the meaning of specific wordings was discussed as a challenge, e.g., the word 'concentration' can be a quantitative measure or a mental concept. 41Numbers were also described as challenging due to ambiguity, because they can refer to the total number of participants, number per arm of a trial, or can just refer to an outcome-related number. 84,113When classifying participants, the P entity or sentence is often overloaded because it includes too much information on different, smaller, entities within it, such as age, gender, or diagnosis. 89biguity in relation-extraction can include cases where interventions and comparators are classified separately in a trial with more than two arms, thus leading to an increased complexity in correctly grouping and extracting data for each separate comparison.

Variations in text
Variations in natural language, wording, or grammar were identified as challenges in many references that looked closer at the texts within their corpora.Such variation may arise when describing entities or sentences (e.g., Refs.48, 79, 97) or may reflect idiosyncrasies specific to one data source, e.g., the position of entities in a specific journal. 46In particular, different styles or expressions were noted as caveats in rule-based systems. 42,48,80ere is considerable variation in how an entity is reported, for example between intervention types (drugs, therapies, routes of application) 56 or in outcome measures. 46In particular, variations in style between structured and unstructured abstracts 65,78 and the description lengths and detail 59,79 can cause inconsistent results in the data extraction, for example by not detecting information correctly or extracting unexpected information.Complex sentence structure was mentioned as a caveat especially for rule-based systems. 80An example of a complex structure is when more than one entity is described (e.g., Refs.93, 102) or when entities such as 'I' and 'O' are mentioned close to each other. 57Finally, different names for the same entity within an abstract are a potential source of problems. 84When using non-English texts, such as Spanish articles, it was noted that mandatory translation of titles can lead to spelling mistakes and translation errors. 35other common variation in text was implied information.For example, rather than stating dosage specifically, a trial text might report dosages of '10 or 20 mg', where the 'mg' unit is implied for the number 10, making it a 'dosage' entity. 46,48,90plied information was also mentioned as problem in the field of relation-extraction, with Nye et al. (2021) 63 discussing importance of correctly matching and resolving intervention arm names that only imply which intervention was used.
Examples are using 'Group 1' instead of referring to the actual intervention name, or implying effects across a group of outcomes, such as all adverse events. 63

Domain adaptation and comparability
Because of the wide variation across medical domains, there is no guarantee that a data extraction system developed on one dataset automatically adapts to produce reliable results across different datasets relating to other domains.The hyperparameter configuration or rule-base used to conceive a system may not retrieve comparable results in a different medical domain. 40,68Therefore, scores might not be similar between different datasets, especially for rule-based classifiers, 80 when datasets are small, 35,49 when structure and distribution of class of interest varies, 40 or when the annotation guidelines vary. 85A model for outcome detection, for example, might learn to be biased towards outcomes frequently appearing in a certain domain, such as chemotherapy-related outcomes for cancer literature or it might favour to detect outcomes more frequent in older trial texts if the underlying training data are older or outdated. 73Another caveat mentioned by Refs.59, 85 is that the size of the label space must be considered when comparing scores, as models that normalise to specific concepts rather than detecting entities tend to have lower precision, recall, and F1 scores.
Comparability between models might be further decreased by comparing results between publications that use relaxed vs. strict evaluation approaches for token-based evaluation, 34 or publications that use the same dataset but with different random seeds to split training and testing data. 33,118erefore, several publications discuss that a larger amount of benchmarking datasets with standardised splits for train, development, and evaluation datasets and standardised evaluation scripts could increase the comparability between published systems. 46,92,1143.5 Computational or system architecture implications Computational cost and scalability were described in two publications. 53,114Problems within the system, e.g., encoding 97 or PDF extraction errors 75 lead to problems downstream and ultimately result in bias, favouring articles from big publishers with better formatted data. 75Similarly, grammar and parsing part-of-speech and/or chunking errors (Refs.76, 80, 90, among others) or faulty parse-trees 78 can reduce a system's performance if it relies on access to correct grammatical structure.In terms of system evaluation, 10-fold cross-validation causes high variance in results when using small datasets such as NICTA-PIBOSO, 54,85,104 described that the same problem needs to be addressed through stratification of the positive instances of each class within folds.

Missing information in text or knowledge base
Information in text can be incomplete. 114For example, the number of patients in a study might not be explicitly reported, 76 or abstracts lacking information about study design and methods can appear, especially in unstructured abstracts and older trial texts. 91,96In some cases, abstracts can be missing entirely.These problems can sometimes be solved by considering the use of full texts as input. 71,87ere a model relies on features, e.g., MetaMap, then missing UMLS coverage causes errors. 72,76This also applies to models like CNNs that assign specific concepts, where unseen entities are not defined in the output label space. 59 terms of automatic summarisation and relation extraction it was also cautioned that relying on abstracts will lead to a low sensitivity of retrieved information, as not all information of interest may be reported in sufficient detail to allow comprehensive summaries or statements about relationships between interventions and outcomes to be made. 60,63

Practical and other implications
In contrast to the problem of missing information, too much information can also have practical implications.For instance, often there are multiple sentences with each label, of which one is 'key', e.g., the descriptions of inclusion and exclusion criteria often span multiple sentences, and for a data extraction system it can be challenging to work out which sentence is the key sentence.The same problem applies to methods that select and rank the top-n sentences for each data extraction target, where a system risks including too much, or not enough results depending on the amount of sentences that are kept. 46w recall is an important practical implication, 53 especially in entities that appear infrequently in the training data, and are therefore not well represented in the training process of the classification system. 48In other words, an entity such as 'Race' might not be labelled very often is a training corpus, and systematically missed or wrongly classified when the data extraction system is used on new texts.Therefore, human involvement is needed, 86 and scores need to be improved. 41It is challenging to find the best set of hyperparameters 106 and to adjust precision and recall trade-offs to maximise the utility of a system while being transparent about the number of data points that might be missed when increasing system precision to save work for a human reviewer. 69,95,101r relation extraction or normalisation tasks, error-propagation was noted as a practical issue in joint models. 63,67To extract relations, first a model to identify entities is needed, and then another model to classify relationships is applied in a pipeline.Neither human nor machine can instantly perform perfect data extraction or labelling, 37 and thus errors done in earlier classification steps can be carried forwards and accumulate.
For relation extraction and summarisation, the importance of qualitative real-world evaluation was discussed.This was due to missing clarity of how well summarisation metrics relate to the actual usefulness or completeness of a summary and because challenges such as contradictions or negations within and between trial texts need to be evaluated within the context of a review and not just a trial itself. 61,63separate practical caveat with relation-extraction models are longer dependencies, i.e. bigger gaps between salient pieces of information in text that lead to a conclusion.This leads to increased complexity of the task and thus to reduced performance. 99 their statement on ethical concerns, DeYoung et al. (2021) 61 mention that these complex relation and summarisation models can produce correct-looking but factually incorrect statements and are risky to be applied in practice without extra caution.

Explainability and interpretability of data extraction systems
The neural networks or machine-learning models from publications included in this review learn to classify and extract data by adjusting numerical weights and by applying mathematical functions to these sets of weights.The decisionmaking process behind the classification of a sentence or an entity is therefore comparable with a black box, because it is very hard to comprehend how, or why the model made its predictions.A recent comment published in Nature has called for a more in-depth analysis and explanation of the decision-making process within neural networks. 117Ultimately, hidden tendencies in the training data can influence the decision-making processes of a data extraction model in a nontransparent way.Many of the examples discussed in the comment are related to healthcare, but in practice there is a very limited understanding of their inherent biases despite the broad application of machine learning and neural networks. 117deeper understanding of what occurs between data entry and the point of prediction can benefit the general performance of a system, because it uncovers shortcomings in the training process.These shortcomings can be related to the composition of training data (e.g.overrepresentation or underrepresentation of groups), the general system architecture, or to other unintended tendencies in a system's prediction. 119A small number of included publications in the base-review (N = 10) discussed issues related to hidden variables as part of an extensive error analysis (see section 3.5.2).The composition of training and testing data were described in most publications, but no publication that specifically addresses the issues of interpretability or explainability was found.

Availability of corpora, and copyright issues
There are several corpora described in the literature, many with manual gold-standard labels (see Table 4).There are still publications with custom, unshared datasets.Possible reasons for this are concerns over copyright, or malfunctioning download links from websites mentioned in older publications.Ideally, data extraction algorithms should be evaluated on different datasets in order to detect over-fitting, to test how the systems react to data from different domains and different annotators, and to enable the comparison of systems in a reliable way.As a supplement to this manuscript, we have collected links to datasets in Table 4 and encourage researchers to share their automatically or manually annotated labels and texts so that other researchers may use them for development and evaluation of new data extraction systems.

Latest developments and upcoming research
This is a timely LSR update, since it has a cut-off just before a the arrival of a new generation of tools: generative 'Large Language Models' (LLMs), such as ChatGPT from OpenAI, based on the GPT-3.5 model [1]. 120As such, it may mark the current state of the field at the end of a challenging period of investigation, where the limitations of recent machine learning approaches have been apparent, and the automation of data extraction was quite limited.
The arrival of transformer-based methods in 2018 marked the last big change in the field, as documented by this LSR.Methods of our included papers only rarely progressed beyond the original BERT architecture, 14 varying mostly just in terms of datasets used in pre-training.Few used models only marginally different to BERT, such as RoBERTa with its altered pre-training strategy. 121However, Figure 13 (reproduced from Yang et al. (2023) 122 ) shows that there has been a vast amount of NLP research and whole families of new methods that have not yet been tested to advance our target task of data extraction.For example within the new GPT-4 technical report, OpenAI describe increased performance, predictability, and closer adherence to the expected behaviour of their model, 123 and some other (open-source) LLMs shown in Figure 13 may have similar potential.
Early evaluations of LLMs suggest that these models may produce a step-change in both the accuracy and the efficiency of automated information extraction, while in parallel reducing the need for expensive labelled training data: a pre-print by Shaib et al. 124 describes a new dataset [2] and an evaluation of GPT-3-produced RCT summaries; 124 Wadhwa, DeYoung, et al. 125 use the Evidence Inference dataset and it's annotations of RCT intervention-comparator-outcome triplets to train and evaluate BRAN, DyGIE++, ELI, BART, T5-base, and several FLAN models in a pre-print; 125 and in a separate pre-print Wadhwa, Amir, et al. 126 used the Flan-T5 and GPT-3 models to extract and predict relations between drugs and adverse events. 126In the near future we expect the number of studies in this review to grow, as more evaluations of LLMs move into pre-print or published literature.

Limitations of this living review
This review focused on data extraction from reports of clinical trials and epidemiological research.This mostly includes data extraction from reports of randomised controlled trials where intervention and comparators are usually jointly extracted, and only a very small fraction of the evidence that addresses other important study types (e.g., diagnostic accuracy studies).During screening we excluded all publications related to clinical data (such as electronic health records) and publications extracting disease, population, or intervention data from genetic and biological research.There is a wealth of evidence and potential training and evaluation data in these publications, but it was not feasible to include them in the living review.

Conclusion
This LSR presents an overview of the data-extraction literature of interest to different types of systematic review.We included a broad evidence base of publications describing data extraction for interventional systematic reviews (focusing on P, IC, and O classes and RCT data), and a very small number of publications extracting epidemiological and diagnostic accuracy data.Within the LSR update we identified research trends such as the emergence of relation-extraction methods, the current dominance of transformer neural networks, or increased code and dataset availability between 2020-2022.However, the number of accessible tools that can help systematic reviewers with data extraction is still very low.
Currently, only around one in ten publications is linked to a usable tool or describes an ongoing implementation.
The data extraction algorithms and the characteristics of the data they were trained and evaluated on were well reported.Around three in ten publications made their datasets available to the public, and more than half of all included publications reported training or evaluating on these datasets.Unfortunately, usage of different evaluation scripts, different methods for averaging of results, or custom adaptations to datasets still make it difficult to draw conclusions on which is the best performing system.Additionally, data extraction is a very hard task.It usually requires conflict resolution between expert systematic reviewers when done manually, and consequently creates problems when creating the gold standards used for training and evaluation of the algorithms in this review.
We listed many ongoing challenges in the field of data extraction for systematic review (semi) automation, including ambiguity in clinical trial texts, incomplete data, and previously unseen data.With this living review we aim to review the literature continuously as it becomes available.Therefore, the most current review version, along with the number of abstracts screened and included after the publication of this review iteration, is available on our website.

Data availability
Underlying data Harvard Dataverse: Appendix for base review.https://doi.org/10.7910/DVN/LNGCOQ. 127is project contains the following underlying data: • Appendix_A.zip (full database with all data extraction and other fields for base review data) • Appendix B.docx (further information about excluded publications) • Appendix_C.zip (code, weights, data, scores of abstract classifiers for Web of Science content) • Appendix_D.zip (full database with all data extraction and other fields for LSR update) • Supplementary_key_items.docx (overview of items extracted for each included study) • included.risand background.ris(literature references from base review) Harvard Dataverse: Available datasets for SR automation.https://doi.org/10.7910/DVN/0XTV25. 128is project contains the following underlying data: • Datasets shared by authors of the included publications Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Extended data
Open Science Framework: Data Extraction Methods for Systematic Review (semi)Automation: A Living Review Protocol.https://doi.org/10.17605/OSF.IO/ECB3T. 15is project contains the following extended data: • Review protocol • Additional_Fields.docx (overview of data fields of interest for text mining in clinical trials) • Search.docx(additional information about the searches, including full search strategies) • PRISMA P checklist for 'Data extraction methods for systematic review (semi)automation: A living review protocol.'Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Reporting guidelines
Harvard Dataverse: PRISMA checklist for 'Data extraction methods for systematic review (semi)automation: A living systematic review' https://doi.org/10.7910/DVN/LNGCOQ. 127ta are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

○
Searching and update schedules have been clearly defined, shown in Figure 1.

○
There are sufficient details of the methods and analysis provided to allow replication.

○
Conclusions are drawn adequately supported by the results presented in the review.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
approaching acceptable methods that are superior to other, less automated methods (the latter of which are not well-discussed).
Some aspects of the paper would benefit from additional detail (in no particular order of importance): The end game for the tracking of this area of literature is not explicitly described in the abstract, nor is it discussed to a great extent at the end of the paper.Much of the results presented do not paint a bright future for this area of research as conditions presently are.While the aim is laid out well in section 1.2, the large amount of missing performance data (reported to be 87%) is unable to address the "Is it reliable?"question.One might suspect that if particularly stellar performance were demonstrated by a project, those data would be prominently advertised.Thus, the yet-to-be-done contacting of authors step would be enlightening if either performance data can be obtained, or if authors remain silent on that request.This follow-up task will be a major point of interest for many who will follow updates to this paper.It is likely that the particular research context (e.g.see Pham et al., 2021 1 ) will have a large degree of influence on the performance metrics to be had if they can be determined.

1.
The description of how the 17 "Key items of interest" were determined and if there is a plan to put these forth as methodological guidelines or a reporting checklist would be helpful.
Either of these would help to advance the field further.

2.
On Page 5, the exclusions listed have the use of pre-processing of text, yet the results discuss the many papers that appear to have used that in their methods.Perhaps this is a deviation from the original protocol after the review began (an understandable decision)? 3.
In section 2.4 about searching Pubmed, can the authors clarify that the Pubmed 2.0 API or GUI will be used to access candidate literature?

4.
Also relevant to section 2.4 on searching, since GITHUB is so popular, might this also be a fruitful place to routinely search?

5.
Clarification of the ability to obtain cited software packages (whether for no cost or at some cost) would be helpful.
Table 5 is shown before Table 1.Please check and correct flow and references to table numbers (5,1,4,2,3 is the flow now).

8.
One of the major limitations to be noted is the unfortunate issue of the lack of specific data in abstracts about interventions and comparators.9.

Is the living method justified? Yes
Have the search and update schedule been clearly defined and justified?Yes Are the rationale for, and objectives of, the Systematic Review clearly stated?Yes Are sufficient details of the methods and analysis provided to allow replication by others?Yes

Are the conclusions drawn adequately supported by the results presented in the review? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Systematic reviews in biomedicine topics, issues with time and effort required to complete reviews with generally available tools.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
whether that has an impact on any of the metrics measured.
In the discussion section, it's interesting that fewer studies extracted data from the full text.Could the authors comment on the implications of this in terms of using tools in a live review as it's not common to manually only extract data from an abstract.

Is the living method justified? Yes
Have the search and update schedule been clearly defined and justified?Yes I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Figure 1 .
Figure 1.Continuous updating of the living review.This image is reproduced under the terms of a Creative Commons Attribution 4.0 International license (CC-BY 4.0) from Schmidt et al.15

Figure 4 .
Figure 4.The most common assessment metrics used in the included publications in order to evaluate the performance of a data extraction system.More than one metric per publication is possible, which means that the total number of included publications (n = 76) is lower than the sum of counts of the bars within this figure.AUC-ROC, area under the curve -receiver operator characteristics; F1, harmonic mean of precision and recall.

Figure 5 .
Figure 5.The study types from which data were extracted.Commonly, randomized controlled trials (RCT) text was at least one of the target text types used in the included publications.

Figure 6 .
Figure 6.The most common entities, as extracted in the included publications.More than one entity type per publication is common, which means that the total number of included publications (n = 76) is lower than the sum of counts within this figure.P, population; I, intervention; C, comparison; O, outcome.

Figure 7 .
Figure 7. Bar chart, showing the levels of algorithm description in the included publications.

Figure 8 .
Figure 8.This chart shows the extent to which included publications provided access to their source code.

3. 4 . 4
Availability of the final model or tool 3.4.4.1 Can we obtain a runnable version of the software based on the information in the publication?

Figure 9 .
Figure 9. Reporting of basic metrics (true positive, false positive, true negative, and false negative).For each included paper.More than one selection is possible, which means that the total number of included publications (n=76) is lower than the sum of counts within this figure.

3. 4 . 5
Internal and external validity of the model 3.4.5.1 Does the dataset or assessment measure provide a possibility to compare to other tools in the same domain?

4 Discussion 4 . 1
Summary of key findings 4.1.1System architectures Systems described within the included publications are changing over time.Non-machine-learning data extraction via rule-base and API is one of the earliest and most frequently used approaches.Various classical machine-learning classifiers such as naïve Bayes and SVMs are very common in the literature published between 2005-2018.Up until 2020 there was a trend towards word embeddings and neural networks such as LSTMs.Between 2020 and 2022 we observed a trend towards transformers, especially the BERT, RoBERTa and ELECTRA architectures pre-trained on biomedical or scientific text.

Figure 11 .
Figure 11.Declaration of funding sources and conflict of interest in the included studies.

Figure 13 .
Figure 13.The evolutionary tree of language models, reproduced from Yang et al. 122 as published in their paper 'Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond'.

○A○
minor consideration is suggested: An incomplete sentence in Methods: 'We included reports published from 2005 until the present day, similar to'.Is the living method justified?YesHave the search and update schedule been clearly defined and justified?YesAre the rationale for, and objectives of, the Systematic Review clearly stated?YesAre sufficient details of the methods and analysis provided to allow replication by others?YesIs the statistical analysis and its interpretation appropriate?Not applicableAre the conclusions drawn adequately supported by the results presented in the review?YesCompeting Interests: No competing interests were disclosed.
Are the rationale for, and objectives of, the Systematic Review clearly stated?YesAre sufficient details of the methods and analysis provided to allow replication by others?PartlyIs the statistical analysis and its interpretation appropriate?Not applicableAre the conclusions drawn adequately supported by the results presented in the review?YesCompeting Interests: No competing interests were disclosed.Reviewer Expertise: Evidence-based medicine, systematic reviews, automation techniques. 15 127e-bases, including approaches using heuristics, wordlists, and regular expressions, were one of the earliest techniques used for data extraction in EBM literature.It remains the most frequently used approaches to automation.Nine publications (12%) use rule-bases alone, while the rest of the publications use them in combination with other classifiers (data shown in Underlying data: Appendix A and D127 3. System architectures used for automating data extraction in the included publications.Results are divided into different categories of machine-learning and natural language processing approaches and coloured by the year of publication.More than one architecture component per publication is possible.Where API, application programming interface; BERT, bidirectional encoder representations from Transformers; CNN, convolutional neural network; CRF, conditional random fields; LSTM, long short-term memory; PICO, population, intervention, comparison, outcome; RNN, recurrent neural networks; SVM, support vector machines. 47licoglu et al. (2021),38Kiritchenko et al. (2010),46Fiszman et al. (2007)47whereasKarystianis et al. (2014, 2017)

Table 1 .
Pre-processing techniques, a short description and examples from the literature.
Concept taggingProcessing and tagging words with semantic classes or concepts, e.g. using word lists or MetaMap75,79,94

Table 3 .
Publications that provide user interfaces to their final data extraction system.

Table 4 .
Corpora used in the included publications.RCT, randomized controlled trials; IR, information retrieval; PICO, population, intervention, comparison, outcome; UMLS, unified medical language system.

Table 4 .
Continued ).For example Ref. 56 used cross-validation to train and evaluate their model, and then used an additional corpus after the cross-validation process.Similarly Ref. 59, used 60:40 train/test splits, but then created an additional corpus of 88 documents to further validate the model's performance on previously unseen data.3.4.5.5 Is the model's adaptability to different formats and/or environments beyond training and testing data described? 46,58,69,48,95,101,102

Table 5 .
Examples for reports of inter-annotator disagreements in the included publications.Please see each included publication for further details on corpus quality.
table 1. csv and table 1_long.csv(Table A1 in csv format, the long version includes extra data) • table 1_long_updated.csv(LSR update for Table A1 in csv format, the long version includes extra data)